System and method for dynamically sharing media in a computer network

ABSTRACT

A computerized method for sharing removable storage media in a network, the method comprising associating, in an index entry, a first piece of removable storage media in a first storage device with at least a first storage policy copy and a second storage policy copy; copying, to the first piece of removable storage media, data associated with the first storage policy copy; and copying, to the first piece of removable storage media, data associated with the second storage policy copy.

RELATED APPLICATIONS

This application claims priority to Provisional Application No.60/460,234, titled SYSTEM AND METHOD FOR PERFORMING STORAGE OPERATIONSIN A COMPUTER NETWORK, filed Apr. 3, 2003 which is hereby incorporatedherein by reference in its entirety.

This application is also related to the following patents and pendingapplications, each of which is hereby incorporated herein by referencein its entirety:

-   -   U.S. Pat. No. 6,418,478, titled PIPELINED HIGH SPEED DATA        TRANSFER MECHANISM, issued Jul. 9, 2002,    -   application Ser. No. 09/610,738, titled MODULAR BACKUP AND        RETRIEVAL SYSTEM USED IN CONJUNCTION WITH A STORAGE AREA        NETWORK, filed Jul. 6, 2000, attorney    -   application Ser. No. 09/744,268, titled LOGICAL VIEW AND ACCESS        TO PHYSICAL STORAGE IN MODULAR DATA AND STORAGE MANAGEMENT        SYSTEM, filed Jan. 30, 2001, and    -   Application Ser. No. 60/409,183, titled DYNAMIC STORAGE DEVICE        POOLING IN A COMPUTER SYSTEM, filed Sep. 9, 2002.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosures, as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightswhatsoever.

BACKGROUND OF THE INVENTION

The invention disclosed herein relates generally to performing storageoperations on electronic data in a computer network. More particularly,the present invention relates to selecting, in response to theinitiation of a storage operation and according to selection logic, amedia management component and a network storage device to performstorage operations on electronic data.

Storage of electronic data has evolved through many forms. During theearly development of the computer, storage of this data was limited toindividual computers. Electronic data was stored in the Random AccessMemory (RAM) or some other storage medium such as a hard drive or tapedrive that was an actual part of the individual computer.

Later, with the advent of networked computing, storage of electronicdata gradually migrated from the individual computer to stand-alonestorage devices accessible via a network. These individual networkstorage devices soon evolved in the form of networkable tape drives,optical libraries, Redundant Arrays of Inexpensive Disks (RAID), CD-ROMjukeboxes, and other devices. Common architectures included drive pools,which generally are logical collections of drives with associated mediagroups including the tapes or other storage media used by a given drivepool.

Serial, parallel, Small Computer System Interface (SCSI), or othercables, directly connected these stand-alone storage devices toindividual computers that were part of a network of other computers suchas a Local Area Network (LAN) or a Wide Area Network (WAN). Eachindividual computer on the network controlled the storage devices thatwere physically attached to that computer and could also access thestorage devices of the other network computers to perform backups,transaction processing, file sharing, and other storage-relatedoperations.

Network Attached Storage (NAS) is another storage scheme usingstand-alone storage devices in a LAN or other such network. In NAS, astorage controller computer still “owns” the storage device to theexclusion of other computers on the network, but the SCSI or othercabling directly connecting that storage device to the individualcontroller or owner computer is eliminated. Instead, storage devices aredirectly attached to the network itself.

Yet another network storage scheme is modular storage architecture whichis more fully described in application Ser. No. 09/610,738 andapplication Ser. No. 09/744,268. An example of such a softwareapplication is the Galaxy™ system, by CommVault Systems of Oceanport,N.J. The Galaxy™ system is a multi-tiered storage management solutionwhich includes, among other components, a storage manager, one or moremedia agents, and one or more storage devices. The storage managerdirects storage operations of client data to storage devices suchmagnetic and optical media libraries. Media agents are storagecontroller computers that serve as intermediary devices managing theflow of data from client information stores to individual storagedevices. Each storage device is uniquely associated with a particularmedia agent and this association is tracked by the storage manager.

A common feature shared by all of the above-described networkarchitectures is the static relationship between storage controllercomputers and storage devices. In these traditional networkarchitectures, storage devices can each only be connected, virtually orphysically, to a single storage controller computer. Only the storagecontroller computer to which a particular device is physically connectedhas read/write access to that device. A drive pool and its associatedmedia group, for example, can only be controlled by the computer towhich it is directly connected. Therefore, all backup from other storagecontroller computers needs to be sent via the network before it can bestored on the storage device connected to the first storage controllercomputer.

At times, storage solutions in some of the above-described networkarchitectures including LAN, NAS, and modular storage systems may causeoverloading of network traffic during certain operations associated withuse of storage devices on the network. The network cable has a limitedamount of bandwidth that must be shared among all the computers on thenetwork. The capacity of most LAN or network cabling is measured inmegabits per second (mbps) with 10 mbps and 100 mbps being standard.During common operations such as system backups, transaction processing,file copies, and other similar operations, network traffic often becomesoverloaded as hundreds of megabytes (MB) and gigabytes (GB) ofinformation are sent over the network to the associated storage devices.The capacity of the network computers to stream data over the network tothe associated storage devices in this manner is greater than thebandwidth capacity of the cabling itself so ordinary network activityand communication slows to a crawl. As long as the storage devices areattached to the LAN or other network, this bandwidth issue remains aproblem.

The Storage Area Network (SAN) is a highly-evolved network architecturedesigned to facilitate transport of electronic data and address thisbandwidth issue. SAN architecture requires at least two networks. First,there is the traditional network described above which is typically aLAN or other such network designed to transport ordinary traffic betweennetwork computers. Then, there is the SAN itself which is a secondnetwork that is attached to the servers of the first network. The SAN isa separate network generally reserved for bandwidth-intensive operationssuch as backups, transaction processing, and the like also describedabove. The cabling used in the SAN is usually of much higher bandwidthcapacity than that used in the first network such as the LAN and thecommunication protocols used over the SAN cabling are optimized forbandwidth-intensive traffic. Most importantly, the storage devices usedby the network computers for the bandwidth-intensive operations areattached to the SAN rather than the LAN. Thus, when thebandwidth-intensive operations are required, they take place over theSAN and the LAN remains unaffected.

CommVault's proprietary DataPipe™ mechanism further described in U.S.Pat. No. 6,418,478 is used with a SAN to further reduce bandwidthconstraints. The DataPipe™ is the transport protocol used to facilitateand optimize electronic data transfers taking place over a Storage AreaNetwork (SAN) as opposed to those taking place over a LAN using NAS.

None of these solutions, however, address the static relationshipbetween individual storage controller computers and individual storagedevices. LANs, WANs, and even SANs using a DataPipe™ all require astatic relationship between storage controller computer and storagedevice since each storage device on the network is uniquely owned by astorage controller computer. As discussed, when a storage device in thistraditional architecture is assigned to a storage controller computer,that storage controller computer owns the device indefinitely and to theexclusion of other computers on the network. This is also true with bothlogical and physical storage volumes. One computer cannot control thedrive pool and media group being that is controlled by another. Requeststo store and retrieve data from such a drive pool and media group wouldhave to first pass through the controlling computer. Such a staticrelationship between storage controller computer and storage deviceoften leads to an inefficient use of resources.

For example, if each storage controller computer needs access to twostorage devices and there are five storage controller computers in thenetwork, then a total of ten storage devices will be required. Theactual amount of work each of the ten storage devices performs might bemuch less than the workload capacity of each storage device. Suchunderutilization of storage device resources cannot be solved when astatic relationship is required between storage device and storagecontroller computer.

If the static relationship were dynamic, however, and storage controllercomputers could actually share devices, then this underutilization canbe addressed. Assuming in the above example that each of the fivestorage controller computers only uses ten percent of each device'sworkload capacity, then if all the storage controller computers couldactually share the same two storage devices, eight of the storagedevices could be eliminated without loss of performance or capability.

Furthermore, none of these existing solutions provide access to storagedevices in the event of a storage controller failure. For example, if astorage controller computer were unavailable due to a hardware orsoftware malfunction, then other computers on the network would not beable to access data stored on any storage device associated with thestorage controller computer. Until the storage controller computer wasbrought back online, the data contained on any associated storage devicewould be effectively unrecoverable. If the association between thestorage controller computer and a storage device were not static,however, then another storage controller computer could bypass theunavailable storage controller computer and access the storage device toretrieve the data.

There is thus also a need for a system which enables dynamic associationof storage controller computers and storage devices

SUMMARY OF THE INVENTION

The present invention addresses, among other things, the problemsdiscussed above performing storage operations on electronic data in acomputer network. In accordance with some aspects of the presentinvention, computerized methods are provided for sharing removablestorage media in a network, comprising associating, in an index entry, afirst piece of removable storage media in a first storage device with atleast a first storage policy copy and a second storage policy copy;copying, to the first piece of removable storage media, data associatedwith the first storage policy copy; and copying, to the first piece ofremovable storage media, data associated with the second storage policycopy.

In some embodiments, methods further include associating, in an indexentry, a second piece of removable storage media in a second storagedevice with at least the first storage policy copy and the secondstorage policy copy; copying, to the second piece of removable storagemedia, data associated with the first storage policy copy; and copying,to the first piece of removable storage media, data associated with thesecond storage policy copy. In some embodiments, methods furtherassociate, in an index entry, the first piece of removable storage mediawith a first media group and a second media group; and associate, in anindex entry, the second piece of removable storage media with the firstmedia group and the second media group.

In one embodiment, a system for sharing removable storage media in anetwork comprises: a first storage device containing a first piece ofremovable storage media; an index entry associating the first piece ofremovable storage media with at least a first storage policy copy and asecond storage policy copy; and a processor, wherein the processor isprogrammed to: copy, to the first piece of removable storage media, dataassociated with the first storage policy copy; and copy, to the firstpiece of removable storage media, data associated with the secondstorage policy copy. In varying embodiments, the processor comprises astorage management component processor, a media management componentprocessor, or a media agent processor. In some embodiments, the systemincludes a second storage device containing a second piece of removablestorage media; an index entry associating the second piece of removablestorage media with at least the first storage policy copy and the secondstorage policy copy; and wherein the processor is programmed to: copy,to the second piece of removable storage media, data associated with thefirst storage policy copy; and copy, to the second piece of removablestorage media, data associated with the second storage policy copy.

In some embodiments, the index entry comprises a storage managementcomponent index cache entry, a media management component index cacheentry, or a media agent index cache entry. In some embodiments, thepiece of removable storage media comprises a piece of magnetic storagemedia, a tape, a piece of optical storage media, or a CD-ROM.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is illustrated in the figures of the accompanying drawingswhich are meant to be exemplary and not limiting, in which likereferences are intended to refer to like or corresponding parts, and inwhich:

FIG. 1 is a block diagram of a network architecture for a system toperform storage operations on electronic data in a computer networkaccording to an embodiment of the invention;

FIG. 2 is a block diagram of an exemplary tape library storage devicefor a system to perform storage operations on electronic data in acomputer network according to an embodiment of the invention; and

FIG. 3 is a block diagram of an exemplary magnetic media storage devicefor a system to perform storage operations on electronic data in acomputer network according to an embodiment of the invention;

FIG. 4 is a flow chart of a method for performing storage operations onelectronic data in a computer network according to an embodiment of theinvention;

FIG. 5 is a flow chart of a method to archive electronic data in asystem to perform storage operations on electronic data in a computernetwork according to an embodiment of the invention;

FIG. 6 is a flow chart of a method for restoring or auxiliary copyingelectronic data in a system to perform storage operations on electronicdata in a computer network according to an embodiment of the invention;

FIG. 7 is a flow chart of a method to restore a storage index in asystem to perform storage operations on electronic data in a computernetwork according to an embodiment of the invention; and

FIG. 8 is a flow diagram of a method to perform dynamic volume sharingaccording to one embodiment of the invention.

DETAILED DESCRIPTION

With reference to FIGS. 1 through 7, embodiments of the invention arepresented. FIG. 1 presents a block diagram of a network architecture fora system to perform storage operations on electronic data in a computernetwork according to an embodiment of the invention. As shown, thesystem includes a storage manager 100 and one or more of the following:a client 85, an information store 90, a data agent 95, a media agent105, an index cache 110, and a storage device 115. The system andelements thereof are exemplary of a three-tier backup system such as theCommVault Galaxy backup system, available from CommVault Systems, Inc.of Oceanport, N.J., and further described in application Ser. No.09/610,738 which is incorporated herein by reference in its entirety.

A data agent 95 is generally a software module that is generallyresponsible for archiving, migrating, and recovering data of a clientcomputer 85 stored in an information store 90 or other memory location.Each client computer 85 has at least one data agent 95 and the systemcan support many client computers 85. The system provides a plurality ofdata agents 95 each of which is intended to backup, migrate, and recoverdata associated with a different application. For example, differentindividual data agents 95 may be designed to handle Microsoft Exchangedata, Lotus Notes data, Microsoft Windows 2000 file system data,Microsoft Active Directory Objects data, and other types of data knownin the art.

If a client computer 85 has two or more types of data, one data agent 95is generally required for each data type to archive, migrate, andrestore the client computer 85 data. For example, to backup, migrate,and restore all of the data on a Microsoft Exchange 2000 server, theclient computer 85 would use one Microsoft Exchange 2000 Mailbox dataagent 95 to backup the Exchange 2000 mailboxes, one Microsoft Exchange2000 Database data agent 95 to backup the Exchange 2000 databases, oneMicrosoft Exchange 2000 Public Folder data agent 95 to backup theExchange 2000 Public Folders, and one Microsoft Windows 2000 File Systemdata agent 95 to backup the client computer's 85 file system. These dataagents 95 would be treated as four separate data agents 95 by the systemeven though they reside on the same client computer 85.

The storage manager 100 is generally a software module or applicationthat coordinates and controls the system. The storage manager 100communicates with all elements of the system including client computers85, data agents 95, media agents 105, and storage devices 115, toinitiate and manage system backups, migrations, and recoveries.

A media agent 105 is generally a software module that conducts data, asdirected by the storage manager 100, between the client computer 85 andone or more storage devices 115 such as a tape library, a magnetic mediastorage device, an optical media storage device, or other storagedevice. The media agent 105 is communicatively coupled with and controlsthe storage device 115. For example, the media agent 105 might instructthe storage device 115 to use a robotic arm or other means to load oreject a media cartridge, and to archive, migrate, or restore applicationspecific data. The media agent 105 generally communicates with thestorage device 115 via a local bus such as a SCSI adaptor. In someembodiments, the storage device 115 is communicatively coupled to thedata agent 105 via a Storage Area Network (“SAN”).

Each media agent 105 maintain an index cache 110 which stores index datathe system generates during backup, migration, and restore storageoperations as further described herein. For example, storage operationsfor Microsoft Exchange data generate index data. Index data provides thesystem with an efficient mechanism for locating user files for recoveryoperations. This index data is generally stored with the data backed upto the storage device 115, and the media agent 105 that controls thestorage operation also writes an additional copy of the index data toits index cache 110. The data in the media agent 105 index cache 110 isthus readily available to the system for use in storage operations andother activities without having to be first retrieved from the storagedevice 115.

The storage manager 100 also maintains an index cache 110. Index data isalso used to indicate logical associations between components of thesystem, user preferences, management tasks, and other useful data. Forexample, the storage manager 100 might use its index cache 110 to tracklogical associations between media agents 105 and storage devices 115.

Index caches 110 typically reside on their corresponding storagecomponent's hard disk or other fixed storage device. Like any cache, theindex cache 110 has finite capacity and the amount of index data thatcan be maintained directly corresponds to the size of that portion ofthe disk that is allocated to the index cache 110. In one embodiment,the system manages the index cache 110 on a least recently used (“LRU”)basis as known in the art. When the capacity of the index cache 110 isreached, the system overwrites those files in the index cache 110 thathave been least recently accessed with the new index data. In someembodiments, before data in the index cache 110 is overwritten, the datais copied to an index cache 110 copy in a storage device 115. If arecovery operation requires data that is no longer stored in the indexcache 110, such as in the case of a cache miss, the system recovers theindex data from the index cache 110 copy stored in the storage device115.

In some embodiments, components of the system may reside and execute onthe same computer. In some embodiments, a client computer 85 componentsuch as a data agent 95, a media agent 105, or a storage manager 100coordinates and directs local archiving, migration, and retrievalapplication functions as further described in application Ser. No.09/610,738. This client computer 85 component can function independentlyor together with other similar client computer 85 components.

FIG. 2 presents a block diagram of an exemplary tape library storagedevice 120 for a system to perform storage operations on electronic datain a computer network according to an embodiment of the invention. Thetape library storage device 120 contains tapes 140 and drives 125, 130,and 135. The tapes 140 store electronic data containing backups ofapplication data, user preferences, system information, and other usefulinformation known in the art. Drives 125, 130, and 135 are used to storeand retrieve electronic data from the tapes 140. In some embodiments,drives 125, 130, and 135 function as a drive pool, as further describedin application Ser. No. 60/409,183 which is hereby incorporated hereinby reference in its entirety. A drive pool is generally a logicalconcept associated with a storage policy. Storage policies representingstorage patterns and preferences are more fully discussed in applicationSer. No. 09/744,268 which is hereby incorporated by reference herein inits entirety.

The drive pool is identified by a set of drives within a tape librarystorage device 120 as pointed to by one or more media agents 105. Forexample, a drive pool known as DP1 consisting of drives 125 and 130 intape library 120 known as LIB1 could be associated by a storage policywith a first media agent 105 MA1 in an index cache 110 entry as follows:LIB1/MA1/DP1. A second drive pool consisting of drives 130, 135, and 140within the tape library storage device 120 associated with the samemedia agent 105 might be expressed in the index cache 110 as follows:LIB1/MA1/DP2.

Previously, however, drive pools had a static association with aparticular media agent 105. A drive pool associated with a first mediaagent 105 could not be logically associated with a second drive poolassociated with a second media agent 105. An index cache 110 entryassociating a drive pool with a media agent 105 and other systemcomponents might, for example, specify a media library or media group, amedia agent, and a drive pool. Only one of each component could bespecified in a particular index cache 110 entry. Thus, such drive poolswere logically exclusive and separate entries in an index cache 110entry for a storage policy and could be logically represented as:

$\left| \frac{{{LIB1}/{MA1}}/{DP1}}{{{LIB1}/{MA2}}/{DP2}} \right.$As further described herein, the present invention permits logicalassociation of drive pools associated with different media agents 105.Multiple drive pools, media agents, and other system components can beassociated in a single index cache 110 entry. Thus, for example, anindex cache 110 entry for a storage policy, according to the presentinvention, may combine the two previous entries instead and thus belogically represented as:

$\begin{matrix}{{{{L1B1}/{MA1}}/{DP1}}/{DP2}} \\{{{{LIB1}/{MA2}}/{DP2}}/{DP1}}\end{matrix}$

In addition and as further described herein, tapes 140 are associated bythe system with drive pools or storage policies, and not with individualdrives 125, 130, and 135. The recording format used to archiveelectronic data is a property of both its media group and its associateddrive pool. A media group is a collection of tapes 140 or other storagemedia assigned to a specific storage policy. The media group dynamicallypoints to different drive pools, even to those with different recordingformats since the system updates the recording format of the media groupin a media group table stored in the index cache 110 of the storagemanager 100. Previously, tapes 140 associated with drive pools couldshare individual drives 125, 130, and 135, but tapes 140 associated witheach drive pool remained logically separate by, among other factors,media groups. Thus, a given set of tapes 140 associated with aparticular drive pool and storing data associated with a first storagepolicy copy could not also store data from a second storage policy copy.An index cache 110 entry would thus associate different tape sets 140with different media agents 105, storage policies, drive pools, andother system components. For example, two different tape sets might beassociated in two index cache 110 entries as follows:

$\left| \frac{{storage}\mspace{14mu}{policy}\mspace{14mu} 1\text{:}\mspace{14mu}{media}\mspace{14mu}{agent}\; 1\text{:}\mspace{14mu}{drive}\mspace{14mu}{pool}\; 1\text{:}\mspace{14mu}{tape}\mspace{14mu}{set1}}{{storage}\mspace{14mu}{policy}\mspace{14mu} 2\text{:}\mspace{14mu}{media}\mspace{14mu}{agent}\; 2\text{:}\mspace{14mu}{drive}\mspace{14mu}{pool}\; 2\text{:}\mspace{14mu}{tape}\mspace{14mu}{set}\; 2} \right.$All components are thus uniquely associated and require separate indexcache 110 entries.

By contrast, the present invention, as further described herein, permitsdata associated with a particular storage policy copy to be stored onand share tapes 140, and other pieces or removable media such as opticaldiscs, associated with and sharing one or more drive pools or storagepolicy copies. Data from each storage policy copy is appended to tapes140 shared by other storage policy copies. Thus, a storage policy copyis shared between many media agents 105 in a dynamic drive poolingsetting with tapes 140 also being shared by the different media agents105 and storage policies. Tapes 140 can be located in any storage device115 and tapes 140 for a given storage policy copy can even be spreadacross multiple storage devices 115. Thus, an index cache entry wouldassociate multiple tape sets 140 with multiple media agents, storagepolicies, drive pools, and other system components. For example, twodifferent tape sets from the previous example of index entries might beassociated in a single index cache 110 entry as follows:

$\begin{matrix}{{storage}\mspace{14mu}{policy}\mspace{14mu} 1\text{:}\mspace{14mu}{media}\mspace{14mu}{agent}\; 1\text{:}\mspace{14mu}{drive}\mspace{14mu}{pool}\; 1\text{:}\mspace{14mu}{tape}\mspace{14mu}{set1}} \\{{storage}\mspace{14mu}{policy}\mspace{14mu} 2\text{:}\mspace{14mu}{media}\mspace{14mu}{agent}\; 2\text{:}\mspace{14mu}{drive}\mspace{14mu}{pool}\; 2\text{:}\mspace{14mu}{tape}\mspace{14mu}{set}\; 2}\end{matrix}$In addition to tape sets 140, a single index cache 110 entry can alsospecify and associate multiple media agents 105, storage policies, drivepools, network pathways, and other components.

Similarly, different media agents 105 and storage policies can also beassociated with the same volumes on magnetic media. For example, turningto FIG. 3, a block diagram is presented of an exemplary magnetic mediastorage device for a system to perform storage operations on electronicdata in a computer network according to an embodiment of the invention.A magnetic media storage device 150, such as a hard disk, is shown. Themagnetic media storage device 150 is divided into two volumes 155 and160 which are associated with a first media agent 165 and a second mediaagent 170. Previously, each volume on a magnetic media storage device150 maintained a static relationship with a given media agent 105. Forexample, if the first volume 155 was associated with the first mediaagent 165, then the second media agent 170 would only be able to conductstorage operations with the first volume 155 by passing any associatedelectronic data through the first media agent 165.

The present invention, however, permits media agents 105 to share thesame volume(s) on a given magnetic storage device. Thus, as furtherdescribed herein, a first media agent 105 can perform, on a givenmagnetic media volume, storage operations on data associated with afirst storage policy copy, and a second media agent can perform, on thesame magnetic media volume, storage operations on different dataassociated with a second storage policy copy. For example, media agent165 and 170 can both perform storage operations on the first volume 155or the second volume 160.

FIG. 4 presents a flow chart of a method for performing storageoperations on electronic data in a computer network according to anembodiment of the invention. Selection of desired storage components forstorage operations is performed dynamically. The system initiates astorage operation, step 175, in response to a scheduled procedure or asdirected by a user. For example, the system might initiate a backupoperation or a restore operation at a specific time of day or inresponse to a certain threshold being passed as specified in a storagepolicy. The system selects a media agent 105 according to selectionlogic further described herein, step 180. Some examples of selectionlogic include the ability to conduct a LAN-free storage operation, suchas using a SAN, and the desire to optimize storage operations via loadbalancing. For example, an index entry in the storage manager 100 indexcache 110 might associate certain media agents 105, storage devices 115,or other components with LAN-free storage operations either via userinput, network topology detection algorithms known in the art, or othermethods. As another example, the system might select a free media agent105 to optimize storage operations via load balancing when a defaultmedia agent 105 or other media agent 105 specified in a storage policyis already performing other storage operations or otherwise occupied.The system also selects an appropriate drive pool in a network storagedevice according to selection logic further described herein, step 185.Once the system has selected an appropriate media agent and drive pool,the storage operation is performed, step 190 using the selected storagecomponents.

FIG. 5 presents a flow chart of a method to archive electronic data in asystem to perform storage operations on electronic data in a computernetwork according to an embodiment of the invention. More specifically,FIG. 5 presents a method for dynamically selecting a drive poolaccording to an embodiment of the invention. The system makes a call toreserve one or more archive streams, step 195. Archive streams aregenerally data paths with permit storage operations to be performed inparallel on electronic data. An archive stream generally has aone-to-one correlation with a media group, such as a media agent 105 andstorage device 115.

Thus, according to one embodiment of the invention, the number ofarchive streams allowed on a storage policy is the sum of all drives inall available drive pools. For example, a system with three drive poolscomposed of ten total drives could reserve ten archive streams to copydata to each of the drives simultaneously thus increasing storageefficiency and achieving other useful goals. In one embodiment, thesystem uses the number of drives in a selected drive pool as the defaultnumber of streams to reserve. In an alternate embodiment, the systemuses magnetic storage libraries as storage devices 115 instead of tapelibraries containing drives. Here, the maximum number of streams wouldequal the number of readers and writers on all mount paths to themagnetic storage libraries.

The system selects one or more appropriate storage locations for thedata to be archived, step 200. In some embodiments, a resource managermodule associated with the storage manager 100 selects the storagelocation(s). For example, a resource manager determines the appropriatestorage device 115, media agent 105, and drive pool combination based onthe pool of available storage network components and other criteria.

Additional criteria used in the selection process generally considerimproving efficiency of the storage operation to be performed. Loadbalancing, for example, is one exemplary consideration. The system maycontain a given number of drives, but some of those drives may be usedby other jobs and thus unavailable. The system thus selects from amongavailable drives as one selection criterion. Alternatively, the networkpath to a particular storage component may be experiencing heavy trafficand thus a less trafficked path offering greater bandwidth may beselected as desirable.

Another exemplary selection criterion is whether selection of a givendrive or set of drives would enable LAN-free or auxiliary archiving. Forexample, in a given network, certain drives might be accessible via aSAN or other alternate storage route which would reduce the networktraffic caused by an archiving operation. Preference, is thus given toselection of these drives over drives which would instead increase thenetwork load.

Yet another exemplary selection criterion is in the case of a storagecomponent failover situation. For example, where an entire media agent105 or storage device 115 as a whole is offline or if a certain numberof drives in a storage device are offline, then the system, in someembodiments, dynamically selects an alternate media agent 105 or drivepool to perform storage operations. In some embodiments, the alternatemedia agent 105 or drive pool in the case of failover is specifiedaccording to preferences associated with a storage policy. For example,a storage policy may contain a list of failover candidates or selectionlogic, as described herein, for selecting a storage location to serve asa failover candidate. In some embodiments, the failover candidates areexpressed as a triplet indicating the media agent 105, the storagedevice 115, and, provided the storage device is not a magnetic mediastorage device 150, the drive pool. Thus, the triplet “LIB1/MA2/DP1”might be used to represent a failover candidate storage path using mediaagent 115 MA2, storage device 115 LIB1, and drive pool DP1. In otherembodiments, a user specifies the alternate storage device 115, mediaagent 105, or drive pool directly.

The system reserves the selected storage candidates, step 205, andreturns the storage IDs of successful reservations to the storagemanager 100, step 210. In some embodiments, the storage IDs are uniqueidentifiers enabling components of the system to identify andcommunicate with the storage candidates. For example, in someembodiments, the storage IDs comprise a unique component name coupledwith a network path such as a Uniform Naming Convention (“UNC”) entry.The storage IDs of the successful reservations are used to update amedia group table stored in the storage manager 100 index cache 110 orother similar locations accessible to components of the system, step215. The reserved components are thus accessible to other components ofthe system to perform the archive operation or other storage operations.

FIG. 6 presents a flow chart of a method to restore or auxiliary copyelectronic data in a system to perform storage operations on electronicdata in a computer network according to an embodiment of the invention.A restore or auxiliary copy operation is initiated and the electronicdata to be restored or copied is identified, step 220.

The system locates the media on which the electronic data to be restoredor copied is stored, step 225. Thus, index data stored at the storagemanager 100, a media agent 105, or another location is consulted todetermine the media ID where the archive file of the electronic datastarts. In some embodiments, the system consults a slot map tablecontained in the index cache 110 to determine the media ID where thearchive file of the electronic data starts.

The system determines the library/media agent/drive pool combination forthe source tape storing the electronic data being restored or copied,step 230. Alternatively, the system determines the library/mediaagent/magnetic storage media volume combination for the electronic databeing restored or copied. As previously described, in some embodimentsthe system determines the media agent 105 according to user preferences,selection logic for increasing network efficiency, or other factors. Forexample, if the electronic data to be restored or copied is located on aparticular tape 140 in a given storage device 115, there may be a finiteset of media agents 105 that are associated with the storage device 115due to network pathways, component failures, user preferences, or otherreasons.

The system establishes a network connection to the selected media agent105 and other components, step 235. In some embodiments, the systemestablishes a high speed connection, such as a DataPipe™ connectionmanufactured by CommVault Systems, and further described in U.S. Pat.No. 6,418,478 which is hereby incorporated herein by reference in itsentirety. Electronic data is thus transferred over the establishedconnection from the storage device 115 to its intended destination suchas a network client 85, an information store 90, or other networkcomponent, step 240.

In some embodiments, the entire archive file is generally created by asingle attempt of backup and is contained within and accessible to thesame media agent 105. Thus, a media agent 105 is identified only wheninitially opening the archive file. When media spanning occurs in themiddle of the archive file, however, such as in the case of a filespanning multiple tapes 140 or volumes, the subsequent pieces of mediacontaining the remaining electronic data must be located and requested,step 245. In the case of media spanning, control thus returns to step225 to locate the additional media and continue the storage operation.

In some embodiments, the initially selected media agent 105 is firstqueried to determine whether it is associated with the additional media,and if so, to continue the storage operation. If the initially selectedmedia agent 105 is not associated with the additional media, the systemqueries the other media agents 105 in the network and attempts to locatethe additional media. For example, the system may search index caches110 associated with the system media agents 105 to locate the additionalmedia. Alternatively, if the system cannot locate the additional media,the user is prompted to manually import or otherwise make available themedia holding the additional electronic data required to complete thestorage operation.

FIG. 7 presents a flow chart of a method to restore a storage index in asystem to perform storage operations on electronic data in a computernetwork according to an embodiment of the invention. In larger storagenetworks, scalability problems sometimes occasion the loss of indexcache 110 directories from media agents 105. For example, in a networkwith many media agents 105 and a great deal of storage operations beingperformed, media agents 105 may load and unload different index caches110 depending on the electronic data subject to the storage operationsbeing performed. Such loss of an index cache 110 directory from thememory of a media agent 105 requires that the index cache 110 directorybe reloaded from stored media such as tapes 140 in a storage device.Reloading an index in this manner, however, often places strain on thenetwork drives, and also results in high system resource usage throughthe instantiation of related processes and threads such as, for example,ifind, createindex, pipelines, and other actions. For example, when amedia agent 105 performs a storage operation, the media agent indexcache 110 is updated using a createindex process to indicate any newnetwork pathways, changes to the files system, version information, andother information useful in performing storage operations. The indexcache 110 directory must first be restored before the createindexprocess can be run to reflect these changes. The present invention, thusprovides, in some embodiments, a method for efficiently restoring anindex cache 110 directory.

The system determines whether there is an index in the cache 110 of aselected media agent, step 255. The system may query media agents 105directly to determine if there is a copy of the index in a local cache110. Alternatively, the system may look for the index in otherlocations, such as in a shared index system as further described herein.

If the index is not in the cache 110, then the system determines themedia ID of the archive file containing the index by consultingreference tables stored in the media agent 105 or the storage manager100 index cache 110, step 260. These tables identify the location of theindex file and identify the actual storage media holding the index filesuch as a particular tape 140, volume 155, or other storage media. Inthe case of a tape 140 or other similar removable storage media, thesystem reserves a drive for accessing the media, step 265, and performsa storage operation to restore the index to the local cache 110 of theselected media agent 105, step 270. In some embodiments, such as in thecase of multiple media agents 105 having access to the index, preferenceis given to a LAN-free or otherwise preferred media agent 105. This isadvantageous in the case of backups since the createindex has a highchance of being on the same machine as the tail of the pipeline.

Once the system determines that the index is in the local cache 110 ofthe selected media agent 105, the index is updated or otherwiseaccessed, such as through a createindex process, to perform the storageoperation at hand, step 275. The storage group table is updated toreflect the storage operations performed, step 280.

In some embodiments, the system employs a shared index cache 110location. A shared index might be employed, for example, to make theindex available to all media agents 105 that may need to participate ina storage operation. Multiple media agents 105, for example, might becandidates for load balancing or failover operations, and all needaccess to the network path of the index so that the index will beavailable to each of them. Otherwise, extra index restores would berequired when different media agents 105 were used for subsequentstorage operations. Thus, in some embodiments, the index cache locationis indicated by a UNC path universally accessible via a username andpassword.

Each media agent 105 is also associated with a unique username andpassword that permit component authentication, access control, and othersimilar functions. The username, password, and UNC path to the sharedindex location is stored in memory associated with each media agent 105.In some embodiments, an indexing module of the media agent 105 employsuser impersonation before accessing the index cache.

In an alternate embodiment, the system employs a shared index cache 110in which a pool of network UNC paths is designated for each media agent105 as a secondary storage area for the index. At the end of eachbackup, a media agent 105 copies the index to this secondary area whichis accessible to all media agents 105 in the pool. Thus, when a mediaagent 105 requires the index, the media agent 105 queries both its localcache 110 and the pool of UNC paths to locate the correct index. If theindex is located in the secondary area and not in the local cache 105,then the index is copied to the local area for immediate use. Uponcompletion of a storage operation, such as a backup or a restore, theindex is copied back to the shared area so that other media agents 105and processes can access the index.

In some embodiments, the system determines which media agent 105, for agiven browse of a client 85 at a point in time, is most likely toalready to have a useable index version already in its local cache 110.For example, in some embodiments, the storage manager 100 tracks whichmedia agent 105 performed the last storage operation for a particularclient 85. A media agent 105 is selected for a client 85 browse at timeT such that the last backup in the full backup cycle at time>=T was donewith indexing at that media agent 105.

FIG. 8 presents a flow diagram showing how dynamic volume sharing isaccomplished according to one embodiment of the invention. A clientapplication or other application initiates a request to the storagemanager 100 to perform a storage operation on electronic data on astorage device 115, such as a magnetic media storage device 150, in thenetwork, and the storage manager 100 processes this request byrequesting access to the volume on which the data is storage, step 285.When a client computer 85 is configured, client data that is to besubject to storage operations is associated with a particular mediaagent 115. When that client data is stored or retrieved in the future,the client computer 85 passes storage operation requests on to theassociated media agent 115. The media agent 115 associates this clientdata with a particular storage media, such as a volume on a magneticmedia storage device 150. Using dynamic volume sharing, one or moremedia agents can store and retrieve data among multiple volumes spanningmultiple magnetic media storage devices 150.

When the media sharing request is received, the storage manager 100verifies that a storage device 115 is available that can be switched toaccommodate the request, step 290. The storage manager 100 tracksstorage device 115 availability in the storage manager index cache 110populated with information regarding available storage devices 115 andtheir respective media agent 105 controllers. Access paths across thenetwork to media agents 105 and then on to appurtenant storage devices115 are also stored in the storage manager index cache 110.

Upon identifying an appropriate storage device 115, the storage manager100 directs the media agent 105 controlling the storage device 115 to gointo a deactivated state with respect to that storage device, step 295.Even though, in some embodiments, there are multiple media agentsexecuting on various hosts for the same storage device 115, therelationship is static and only one of them can control a storage device115 at a given instant. The other media agents 105 are said to be in adeactivated state with respect to that storage device 115.

The deactivated media agents 105 run a listening process waiting for amessage from the storage manager 100 directing them to become activewith respect to a storage device 115. Once the first media agent 105 hasbeen deactivated with respect to the storage device 115, the storagemanager communicates to the listening process of a second media agent105 on which the storage device 115 will be mounted to change from adeactivated state to an activated state with respect to the storagedevice 115, step 300. At this point the storage manger 100 also updatesthe storage manager cache 110 to reflect that control of the storagedevice 115 has been shifted from the first media agent 105 to the secondmedia agent 105, and that the first media agent is now deactivated andthat the second media agent is now activated with respect to thatstorage device, step 305.

The second media agent 105 communicates with the storage device 115 andexecutes procedures necessary to mount storage device 115 and anyassociated volumes to the second media agent, step 310. In someembodiments, the second media agent 105 mounts one or more of thevolumes associated with the storage device 115, and volumes in the samestorage device 115 not mounted by the second media agent 105 may bemounted or otherwise associated with other media agents 105. Once themount is performed the storage device 150 and its associated volumes 150are logically connected to the second media agent 105, and this accesspath is stored by the second media agent 105 in its index cache 110,step 315. The media agent 105 stores the access path to the storagedevice 115 in the media agent index cache 110 because a storage device115 connected to multiple media agents 105 may have multiple accesspaths. Mounting the storage device 115 to the media agent 105 and theresultant access path produced is in large part related to the hardwareconfiguration of the media agent 105. The media agent 105 is generallybest-suited to store and delegate management of the access path to thestorage device that it controls. In some alternate embodiments, thestorage manager 100 stores and tracks the individual hardwareconfiguration of all the network media agents 105 in the storage managerindex cache 110 and then passes the resultant access paths to thenetwork storage devices 115 on to the media agents 105 when necessary.In other embodiments, media agent 105 hardware configurations andresultant access paths to the network storage devices 115 are stored ina shared index location as further described herein.

Once the media agent 105 has completed the mount of the storage device115 (and any associated volumes) and stored the access path to thestorage device 115 in its own media agent index cache 110 or otherlocation, then the access path to the storage device 115 is returned bythe media agent 105 to the storage manager 100 where it is also storedin the storage manager index cache 110 for future recall, step 320.While media agents 115 generally communicate with storage devices 115and the storage manager 100, the storage manager 100 generallycommunicates with client applications. In some embodiments, the storagemanager 100 returns the storage device access path to a clientapplication or other application and initiates a storage operation asappropriate, step 325.

Systems and modules described herein may comprise software, firmware,hardware, or any combination(s) of software, firmware, or hardwaresuitable for the purposes described herein. Software and other modulesmay reside on servers, workstations, personal computers, computerizedtablets, PDAs, and other devices suitable for the purposes describedherein. Software and other modules may be accessible via local memory,via a network, via a browser or other application in an ASP context, orvia other means suitable for the purposes described herein. Datastructures described herein may comprise computer files, variables,programming arrays, programming structures, or any electronicinformation storage schemes or methods, or any combinations thereof,suitable for the purposes described herein. User interface elementsdescribed herein may comprise elements from graphical user interfaces,command line interfaces, and other interfaces suitable for the purposesdescribed herein. Screenshots presented and described herein can bedisplayed differently as known in the art to input, access, change,manipulate, modify, alter, and work with information.

While the invention has been described and illustrated in connectionwith preferred embodiments, many variations and modifications as will beevident to those skilled in this art may be made without departing fromthe spirit and scope of the invention, and the invention is thus not tobe limited to the precise details of methodology or construction setforth above as such variations and modification are intended to beincluded within the scope of the invention.

1. A computerized method for sharing removable storage media in anetwork, the method comprising: associating, in an index entry, a firstpiece of removable storage media in a first storage device with at leasta first storage policy copy and a second storage policy copy; copying,to the first piece of removable storage media, data associated with thefirst storage policy copy; and copying, to the first piece of removablestorage media, data associated with the second storage policy copy. 2.The method of claim 1, wherein associating, in an index entry, comprisesassociating in a storage management component index cache entry.
 3. Themethod of claim 1, wherein associating, in an index entry, comprisesassociating in a media management component index cache entry.
 4. Themethod of claim 3, wherein associating in a media management componentindex cache entry comprises associating in a media agent index cacheentry.
 5. The method of claim 1, wherein associating a piece ofremovable storage media comprises associating a piece of magneticstorage media.
 6. The method of claim 5, wherein associating a piece ofmagnetic storage media comprises associating a tape.
 7. The method ofclaim 1, wherein associating a piece of removable storage mediacomprises associating a piece of optical storage media.
 8. The method ofclaim 7, wherein associating a piece of optical storage media comprisesassociating a CD-ROM.
 9. The method of claim 1, comprising: associating,in an index entry, a second piece of removable storage media in a secondstorage device with at least the first storage policy copy and thesecond storage policy copy; copying, to the second piece of removablestorage media, data associated with the first storage policy copy; andcopying, to the first piece of removable storage media, data associatedwith the second storage policy copy.
 10. The method of claim 9,comprising: associating, in an index entry, the first piece of removablestorage media with a first media group and a second media group; andassociating, in an index entry, the second piece of removable storagemedia with the first media group and the second media group.
 11. Themethod of claim 10, wherein associating, in an index entry, comprisesassociating in a storage management component index cache entry.
 12. Themethod of claim 10, wherein associating, in an index entry, comprisesassociating in a media management component index cache entry.
 13. Themethod of claim 12, wherein associating in a media management componentindex cache entry comprises associating in a media agent index cacheentry.
 14. A system for sharing removable storage media in a network,the system comprising: a first storage device containing a first pieceof removable storage media; an index entry associating the first pieceof removable storage media with at least a first storage policy copy anda second storage policy copy; and a processor, wherein the processor isprogrammed to: copy, to the first piece of removable storage media, dataassociated with the first storage policy copy; and copy, to the firstpiece of removable storage media, data associated with the secondstorage policy copy.
 15. The system of claim 14, wherein the index entrycomprises a storage management component index cache entry.
 16. Thesystem of claim 14, wherein the index entry comprises a media managementcomponent index cache entry.
 17. The system of claim 16, wherein themedia management component index entry comprises a media agent indexcache entry.
 18. The system of claim 14, wherein the piece of removablestorage media comprises associating a piece of magnetic storage media.19. The system of claim 18, wherein the a piece of magnetic storagemedia comprises associating a tape.
 20. The system of claim 14, whereinthe piece of removable storage media comprises associating a piece ofoptical storage media.
 21. The system of claim 20, wherein the piece ofoptical storage media comprises associating a CD-ROM.
 22. The system ofclaim 14, wherein the processor comprises a storage management componentprocessor.
 23. The system of claim 14, wherein the processor comprises amedia management component processor.
 24. The system of claim 23,wherein the media management component processor comprises a media agentprocessor.
 25. The system of claim 24, comprising: an index entryassociating the first piece of removable storage media with a firstmedia group and a second media group; and an index entry associating thesecond piece of removable storage media with the first media group andthe second media group.
 26. The system of claim 14, comprising: a secondstorage device containing a second piece of removable storage media; anindex entry associating the second piece of removable storage media withat least the first storage policy copy and the second storage policycopy; and wherein the processor is programmed to: copy, to the secondpiece of removable storage media, data associated with the first storagepolicy copy; and copy, to the second piece of removable storage media,data associated with the second storage policy copy.
 27. The system ofclaim 26, wherein the index entry comprises a storage managementcomponent index cache entry.
 28. The system of claim 27, wherein themedia management component index cache entry comprises a media agentindex cache entry.
 29. The system of claim 26, wherein the index entrycomprises a media management component index cache entry.